Skip to content

Native: opt-in capabilities digest instead of ~17 KB server_capabilities on every worker answer - #1262

Merged
rmcdaniel merged 6 commits into
mainfrom
general/egress-capabilities-digest
Nov 20, 2026
Merged

rmcdaniel merged 6 commits into
mainfrom
general/egress-capabilities-digest

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Why

Every worker-plane JSON answer carries server_capabilities, as PHP's WorkerProtocol::json does (server#1001). A completion that carries a poll carries a second copy inside its answer. On native that object is about 17 KB.

For DW Standard Workflow v1 (one start, one activity, one completion), a worker receives about 18 copies per workflow, roughly 320 KB of the ~385 KB of responses. That traffic was what saturated the Cloud Dev cell's 4 Mbit/s egress at 1.5 wf/s. In the 2 wf/s step, the saturation also cost a workflow its activity lease (sdk-php#445).

Per-endpoint response bytes per workflow (lab, main, 1 wf/s, 4 workers, 1 s task polls):

endpoint calls/wf bytes/wf of which server_capabilities
POST worker/query-tasks/poll (empty) 5.0 85.5 KB 17.1 KB per call
POST worker/workflow-tasks/poll 4.0 77.5 KB 17.1 KB per call
POST worker/activity-tasks/poll (always empty: activities arrive carried) 4.0 68.6 KB 17.1 KB per call
POST worker/workflow-tasks/{id}/complete (+ carried activity poll) 2.0 53.7 KB 2 copies per call
POST worker/activity-tasks/{id}/complete (+ carried workflow poll) 1.0 46.3 KB 2 copies per call
POST worker/heartbeat 0.43 7.5 KB 1 copy per call
client: describe ×2, history ×1, cluster info, start 45.6 KB —
total 384.8 KB ~306 KB

What

This adds an opt-in, backward-compatible digest. The contract is in docs/contracts/server-capabilities-digest.md.

  • A worker sends X-Durable-Workflow-Server-Capabilities-Digest: <digest>|none.
  • If the header matches the current digest, each copy of the current capabilities is replaced by the digest string. That covers the top level and up to two objects deep, as in activity_poll.answer and workflow_poll.answer.
  • If it does not match, each copy is sent in full with a digest member added.
  • Every opted-in answer also carries the current digest in the same header.
  • Requests without the header get the same bytes as before. Existing SDKs are unaffected, including the exact-key validation of carried answers.
  • A copy that differs from the current capabilities is never replaced.
  • Implementation: a separate layer outside the worker-envelope guard, so it sees the completed envelope and raw member text. The members are re-rendered from their raw values, the same way the envelope code does it.
  • Layering: this branch is rebased on Native: compress JSON responses like PHP's CompressResponse (Dev egress 385 → 106 KB/workflow) #1261 and sits inside response compression, which stays the outermost layer. The digest layer reads plain JSON, and compression then encodes the smaller body. If the order were reversed, the digest layer would skip encoded bodies and the savings would silently vanish. A router-level test covers this. Merge Native: compress JSON responses like PHP's CompressResponse (Dev egress 385 → 106 KB/workflow) #1261 first.
  • Per-request resolution: the server replaces copies only with the digest that request sent. A worker therefore resolves a digest string against the digest and object it captured when sending that request, which is safe when requests overlap a capabilities change, such as a rolling deploy. The rule is in the contract and implemented in sdk-php#446.

The SDK side is in durable-workflow/sdk-php (linked below). Its transport restores the held object before any caller sees the answer, covering success bodies, refusals and poll frames.

Measured (lab)

Setup

  • Native build of main, in a 1 vCPU / 1 GiB Dev-like slice.
  • DW Standard Workflow v1, 1 wf/s, 4 workers.
  • A byte meter that behaves like the Cloudflare edge: it asks the origin for br, gzip, and logs origin bytes.
  • Window: 120 s, with all workflows verified.
server SDK origin response bytes / wf
main 3.22.0 384.8 KB
main + #1261 (compression) 3.22.0 106.0 KB
main + this PR, compression off patched 60.1 KB (digest −306 KB, history-free result −18.5 KB)
main + #1261 + this PR patched 18.4 KB

CPU (lab slice, 1 vCPU, including MySQL):

p50 and p95 latency were unchanged (1.04 s / 1.09 s).

Tests

  • Unit (capabilities_digest):
    • Requests without the header are unchanged.
    • With none or a stale digest, the copies are tagged, and the digest covers the untagged object.
    • With a matching digest, both the top-level and carried copies are replaced, and the task and other members are preserved.
    • Values that differ, and control-plane answers, are never touched.
  • Router-level (tests/execution.rs::the_capabilities_digest_is_applied_inside_gzip_compression): a request with Accept-Encoding: gzip and the digest header gets a gzip body whose decoded server_capabilities is the digest string.
  • Local runs:
    • cargo test --lib -- capabilities_digest runtime::http workflow_task_guard carried_polls: 16 passed.
    • cargo test --test execution -- carried: 7 passed.

🤖 Generated with Claude Code

Native sent every answer uncompressed and advertised response_compression: [].
PHP Server gzip- or deflate-encodes JSON answers of 1024 bytes or more when
Accept-Encoding allows it (DW_COMPRESSION_ENABLED, default true).

On a Dev cell behind a Cloudflare tunnel the origin answers are the cell's
egress, which is shaped to 4 Mbit/s. The Cloudflare edge always asks the
origin for "br, gzip" and decodes for clients that did not ask, so this
reduces tunnel egress without any SDK change. Lab measurement (DW Standard
Workflow v1, 1 wf/s, 4 workers): 384.8 KB -> 106.0 KB of origin response
bytes per workflow.

- New response_compression middleware, outermost on the router: PHP's
  encoding preference (gzip over deflate at equal q, *, q=0), JSON content
  types only, 1024-byte minimum, existing Content-Encoding left alone,
  Vary: Accept-Encoding. Only in-memory (exact-sized) bodies are encoded;
  streamed answers pass through.
- DW_COMPRESSION_ENABLED (and its legacy name) is honored instead of ignored.
- cluster info advertises ["gzip", "deflate"] when enabled, as PHP does; the
  native deviation is removed from the parity fixture.
- History-page gzip reuses the same RFC 1952 encoder.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel

Copy link
Copy Markdown
Member Author

SDK side: durable-workflow/sdk-php#446. Compression: #1261.

Review fixes:
- Bodies above 256 KB are encoded on a blocking thread (spawn_blocking).
  At roughly 30 MB/s for level 6, an 8 MB body would otherwise hold an async
  worker for about 300 ms on a 2-thread runtime.
- HTTP's deflate coding is the zlib format (RFC 1950), not raw DEFLATE;
  deflate answers are now zlib-wrapped.
- Tests: a large answer encodes on the blocking path and round-trips for
  gzip and deflate, the zlib header is checked, and an ignored benchmark
  reports encode cost by size.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel and others added 3 commits November 20, 2026 01:56
encode_cost (release, lab): 14 KB in 0.6 ms, 217 KB in 10 ms, 7 MB in
196 ms (22-36 MB/s).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every worker-plane JSON answer carries the ~17 KB server_capabilities
object, and a completion that carries a poll carries a second copy in its
answer. For DW Standard Workflow v1 that is about 18 copies (~320 KB) of
the ~385 KB of responses per workflow, and it dominated a Dev cell's
4 Mbit/s egress.

A worker that sends X-Durable-Workflow-Server-Capabilities-Digest opts in.
When the header names the current digest, each copy of the current
capabilities (top level, or inside a carried poll's answer) is replaced by
the digest string; otherwise each copy is sent in full with a digest
member for the worker to keep. Requests without the header are unchanged,
so existing SDKs see the same bytes. Contract:
docs/contracts/server-capabilities-digest.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review fixes:
- The contract states that the server replaces copies only with the digest
  that request sent, so a worker resolves a digest string against the digest
  and object it captured when sending that request. A worker that kept only
  its latest digest could reject a committed answer during a rolling deploy.
  Version names and PR links are removed from the contract prose.
- A router-level test sends Accept-Encoding: gzip and the digest header and
  asserts the digest is applied inside a gzip-encoded body (compression is
  the outer layer).
- The header is documented in server-reference.md and rust-server-port.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel
rmcdaniel force-pushed the general/egress-capabilities-digest branch from 7087d39 to 93be253 Compare November 20, 2026 01:56
@rmcdaniel

Copy link
Copy Markdown
Member Author

Review fixes pushed. Rebased on #1261 (merge #1261 first); compression stays the outermost layer. Added the router-level gzip+digest test, the per-request resolution rule in the contract (version names and PR links removed), and documentation in server-reference.md and rust-server-port.md.

Local runs: cargo test --lib -- capabilities_digest response_compression: 10 passed. cargo test --test execution -- the_capabilities_digest_is_applied carried activity_results: 12 passed.

Draft 3.17 release-notes items:

Smaller worker and client responses

Native now sends far fewer response bytes. In a lab run of the standard
one-activity workflow, responses fell from about 385 KB to about 18 KB per
workflow, with the matching PHP SDK. Worker protocol and control-plane versions
are unchanged, and no schema step is needed for this.

  • Response compression is on by default. JSON answers of 1024 bytes or more
    are gzip-encoded, or deflate-encoded (zlib format), when the request's
    Accept-Encoding allows it. Native follows PHP Server's CompressResponse:
    gzip wins at equal weight, q=0 refuses an encoding, an answer that already
    has a Content-Encoding is left alone, and Vary: Accept-Encoding is added.
    Bodies over 256 KB are encoded on a blocking thread, and streamed answers pass
    through unchanged. Behind Cloudflare (Cloud Dev and similar tunnels) the edge
    always asks the origin for gzip, so existing clients benefit without any
    change.
  • DW_COMPRESSION_ENABLED is now honoured, with the legacy name
    WORKFLOW_SERVER_COMPRESSION_ENABLED. It defaults to true; set it to
    false to turn compression off, for example when a reverse proxy already
    compresses. It used to be accepted and ignored.
  • response_compression is advertised. Cluster info reports
    capabilities.response_compression: ["gzip", "deflate"] when compression is
    enabled and [] when it is disabled, as PHP Server does.
  • New opt-in header: X-Durable-Workflow-Server-Capabilities-Digest.
    Workers that send it stop receiving the ~17 KB server_capabilities object in
    every worker answer, including the second copy inside carried polls.
    • When the header names the current digest, each copy is replaced by the
      digest string.
    • Otherwise each copy is sent in full with a digest member the worker keeps.
    • A digest string always names the digest that request sent, so a worker
      resolves it against what it captured when it sent that request. This is
      safe across rolling deploys.
    • Workers that don't send the header get exactly the same bytes as before.
    • Contract: docs/contracts/server-capabilities-digest.md. PHP SDK support
      ships separately.

The elided empty poll is under the 1024-byte compression threshold, so the
second answer is now a leased workflow task: still over the threshold with
its capabilities elided, so both layers apply to it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant